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I.  INTRODUCTION 


A.  INTRODUCTION 

The  volume  of  services,  both  in  dollar  amount  and  type,  acquired  by  the 
Department  of  Defense  (DoD)  has  steadily  grown  over  the  past  few  fiscal  years.  In  the 
period  between  fiscal  years  2000  and  2005,  DoD  experienced  a  73-percent  increase  in 
service  contract  costs  in  the  operations  and  maintenance  areas.  The  chief  factors  behind 
the  growth  in  acquired  services  include  1)  the  impact  the  Global  War  on  Terrorism  has 
had  on  the  number  of  requirements  and  the  resulting  increase  in  the  DoD’s  use  of 
contractors  to  meet  those  needs,  2)  the  federal  policy  mandating  reliance  on  contractors 
for  functions  not  inherently  governmental,  3)  competitive  sourcing  and  privatization 
programs  within  the  DoD  and  4)  headquarters  personnel  manning  level  limitations 
leading  to  dependence  on  contractors  to  complete  new  and  expanded  work  requirements. 
(GAO,  2007) 

To  achieve  increased  data  granularity  on  these  primary  factors  and  an  accurate 
weighting  of  each  one’s  impact,  empirical  data  needs  to  be  collected  and  analyzed.  We 
believe  that  data  collection  at  the  installation  level  throughout  DoD  will  provide  the 
necessary  granularity.  Coding  and  analysis  of  this  gathered  data  should  clarify  whether 
the  recent  trends  will  continue  or  abate  and  whether  they  are  “treatable”  or  inherent  to 
modem  day  DoD  services  acquisition.  Once  the  ground  truth  of  the  current  services 
acquisition  situation  is  better  understood,  alternatives  could  be  developed,  and  sound 
services  acquisition  management  policy  recommendations  can  be  forwarded  to  DoD 
leadership  for  consideration  and  implementation. 

Directly  linked  with  the  acquisition  of  services  is  the  management  of  those 

acquired  services.  Apte,  Ferrer,  Lewis,  and  Rendon  (2006)  conducted  exploratory 

research  regarding  the  size,  trends,  and  issues  related  to  services  acquisition.  Data  was 

collected  through  site  visits  to  the  Presidio  of  Monterey,  Travis  Air  Force  Base,  and  the 

Naval  Postgraduate  School.  One  of  their  research  findings  was  that  despite  the  high 

expenditure  levels  for  services,  the  acquisition  management  structure  for  service 
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contracts  is  lacking.  Unlike  the  management  infrastructure  for  a  large  weapon  system 
procurement  program,  which  would  have  a  dedicated  program  manager  and  robust 
management  team,  services  acquisition  procurement  programs  do  not  offer  the  same  level 
of  oversight,  thus  opening  the  door  for  mismanagement. 

Apte  and  Rendon  (2007)  conducted  a  follow-up  research  project  with  a 
concentration  on  the  applicability  of  a  program  management  approach  in  managing 
acquired  services  within  DoD.  The  project  provided  further  discussion  of  issues  in 
services  contracting  including  the  intangibility  of  service  outcomes,  co-production  of 
services  and  the  diversity  of  services.  In  their  research  the  principal  investigators 
discussed  basic  concepts  of  program  management  and  how  those  concepts  are  currently 
applied  to  the  acquisition  of  products.  Using  this  framework,  the  project  analyzed 
services  acquisition  management  at  the  installations  they  visited  in  the  first  project,  in 
addition  to  Randolph  Air  Force  Base  in  Texas.  It  was  concluded  that  though  some 
program  management  concepts  were  being  applied,  they  were  being  applied 
inconsistently  and  did  not  necessarily  build  the  framework  for  a  program  management 
approach  to  services  acquisition. 

B.  PURPOSE 

The  purpose  of  this  project  is  to  determine  how  to  best  collect  empirical  data 
regarding  the  current  state  of  services  acquisition  management  at  the  installation  level 
across  the  military  services  and  to  conduct  an  initial  analysis  of  collected  data.  The 
results  of  this  project  will  support  on-going  research  (Apte  et  ah,  2006,  Apte  and  Rendon, 
2007)  being  conducted  by  the  Acquisition  Research  Program  at  the  Naval  Postgraduate 
School  regarding  the  Department  of  Defense’s  management  of  services  acquisition. 

C.  RESEARCH  QUESTIONS 

There  are  several  research  questions  which  this  project  tries  to  answer  and  they 
are  as  follows. 
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Primary: 

•  What  is  the  current  state  of  services  acquisition  management,  at  the  installation 
level? 

Secondary: 

•  What  research  method  should  be  used  to  best  evaluate  the  current  state  of  services 
acquisition  management,  across  all  military  services,  in  a  uniform  and  unbiased 
manner? 

•  What  is  the  best  way  to  tailor  the  chosen  research  method  to  produce  usable 
results? 

•  How  should  the  chosen  research  method  be  tailored  to  answer  the  primary 
research  question? 

Researching  answers  to  these  questions  will  provide  infonnation  and  data  will 
facilitate  in  studying  the  six  research  questions  posed  by  the  aforementioned  of  on-going 
research.  These  six  research  questions  are  1)  What  types  of  services  are  typically 
contracted  for  at  military  installations  and  what  is  the  annual  expenditure  for  these 
services?  2)  What  types  of  acquisition  strategies,  procurement  methods,  and  contracts  are 
being  used  to  acquire  services?  3)  How  are  these  service  contracts  managed?  4)  What 
types  of  organization/management  structures  are  used  to  manage  contracted  services?;  5) 
What  training  does  contract  and  project/program  management  staff  receive?  and  6)  Do 
the  respective  military  services  acquire  and  manage  services  differently? 

D.  SERVICE  CATEGORIES 

The  Federal  Procurement  Data  System  Product  and  Service  Codes  Manual 
identifies  and  describes  twenty-four  service  categories  used  in  the  Federal  Procurement 
Data  System.  The  types  of  services  provided  for  under  these  categories  range  from 
special  studies  and  analysis  to  utilities  and  housekeeping  services.  Table  1  on  the 
following  page  provides  the  complete  listing  of  service  categories  and  associated  PSC 
codes. 
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Table  1.  Product  Service  Categories  and  Codes 


Service  Category 

Product/Service 
Classification  (PSC)  Code 

Research  and  development 

A 

Special  studies  and  analysis 

B 

Architect  and  engineering  services 

C 

Data  processing  and  telecommunications 

D 

Purchase  of  structures  and  facilities 

E 

Conservation  and  natural  resources 

F 

Social  Services 

G 

Quality  control,  testing  and  inspection  services 

H 

Maintenance  and  repair  of  equipment 

J 

Modification  of  equipment 

K 

Technical  representative  services 

L 

Operation  of  government-owned  facilities 

M 

Installation  of  equipment 

N 

Salvage  services 

P 

Medical 

Q 

Professional,  administrative,  and  management  support 

R 

Utilities  and  housekeeping 

S 

Photographic,  mapping,  printing  and  publication 
services 

T 

Educational  and  training  services 

U 

Transportation  and  travel 

V 

Lease  or  rental  of  equipment 

W 

Lease  or  rental  of  facilities 

X 

Construction  of  structures  and  facilities 

Y 

Maintenance  and  repair  of  real  property 

Z 

E.  PROJECT  SCOPE  LIMITATIONS 

The  shear  volume  of  services  acquisition  within  the  DoD  along  with  other 
limiting  factors  decided  the  scope  of  this  project.  The  first  is  the  reduction  in  the  number 
of  service  categories  considered  to  seven  of  the  twenty-four  mentioned  in  Table  1.  These 
seven  service  categories  were  selected  for  further  examination  based  on  data  presented  in 
the  Government  Accountability  Office’s  (GAO)  May  2007  report,  Defense  Budget: 
Trends  in  Operation  and  Maintenance  Costs  and  Support  Sendees  Contracting  as  shown 
in  Table  2. 
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Table  2.  Changes  in  Service  Contract  Costs  in  Selected  Categories  (From 
GAO  2007) 


Fiscal  year  2007  dollars  in  billions 

Contract  Costs 

Change  from  fiscal 
year  2000  to  2005 

FY 

FY 

Service  category 

2000 

2005  Amount  Percentage 

Professional,  administrative,  and  management  support 

$14.6 

$30.1 

$15.5 

107 

Maintenance  and  repair  of  equipment 

7.7 

12.3 

4.6 

60 

Data  processing  and  telecommunications 

6.3 

11.0 

4.7 

74 

Medical 

2.8 

8.4 

5.6 

199 

Maintenance  and  repair  of  real  property 

6.6 

8.0 

1.5 

22 

Utilities  and  housekeeping 

3.9 

7.0 

3.1 

79 

Transportation  and  travel 

3.4 

6.6 

3.3 

97 

Conservation  and  natural  resources 

1.7 

2.3 

0.7 

39 

Operation  of  government-owned  facilities 

2.3 

2.1 

(0.2) 

(9) 

Technical  representative  services 

1.4 

1.7 

0.3 

23 

Special  studies  and  analyses 

1.2 

1.5 

0.2 

19 

Modification  of  equipment 

1.1 

1.4 

0.3 

29 

Educational  and  training  services 

1.1 

1.4 

0.3 

23 

Other 

1.3 

2.0 

0.7 

58 

Total 

$55.4 

$95.9 

$40.6 

73 

GAO  considered  19  of  the  24  service  categories  in  the  report.  The  seven  service 
categories  selected  for  consideration  within  this  project,  highlighted  in  Table  3  on  the 
next  page,  accounted  for  more  than  $83  billion  in  expenditures  during  fiscal  year  2005, 
up  from  $45.3  billion  in  fiscal  year  2000.  The  $83  billion  spent  on  services  in  fiscal  year 
2005  accounted  for  roughly  87-percent  of  expenditures  on  services.  Additionally,  six  of 
the  seven  selected  categories  of  services  showed  the  greatest  percentage  changed  in 
dollars  spent  between  fiscal  year  2000  and  fiscal  year  2005. 


5 


Table  3.  Service  Categories  Considered  by  Project 


Service  Category 

Product/Service 
Classification  (PSC)  Code 

Professional,  administrative,  and  management  support 

R 

Maintenance  and  repair  of  equipment 

J 

Data  processing  and  telecommunications 

D 

Medical 

Q 

Utilities  and  housekeeping 

S 

Transportation  and  travel 

V 

Maintenance  and  repair  of  real  property 

z 

We  further  limited  our  research  to  installations  within  the  continental  United 
States  (CONUS)  and  eliminated  international  bases.  This  limitation  arose  from 
complications  in  accounting  for  data  considering  varying  currency  exchange  rates, 
economies  and  operational  situations  indicative  of  divergent  foreign  locations,  not  to 
mention  communications  across  time  zones. 

Another  limitation  is  the  number  of  military  services  to  consider.  Research  is 
needed  across  all  four  branches  of  military  service.  However  with  limited  personnel 
resources  and  the  project  team  members’  respective  affiliation  with  the  Army  and  Marine 
Corps,  this  project  is  limited  to  those  military  services’  installations.  All  four  services 
would  be  beyond  the  current  capabilities  of  the  project  team,  but  follow-on  projects  to 
capture  and  analyze  Air  Force  and  Navy  installations  data  are  already  underway. 

F.  METHODOLOGY 

A  survey  methodology  will  be  used  to  facilitate  the  gathering  of  preliminary  data. 
A  survey  will  be  designed  and  issued  to  a  sample  audience  for  conduct  of  the  pilot  test. 
The  qualitative  results  of  the  pilot  test  will  be  used  to  adjust  the  survey  to  increase  survey 
reliability.  The  quantitative  results  of  the  pilot  test  will  be  analyzed  and  preliminary 
observations  regarding  the  acquisition  management  of  services  will  be  provided. 
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G. 


ORGANIZATION 


This  project  is  organized  into  five  chapters.  This  introductory  chapter  is  followed 
by  Chapter  II  which  describes  the  aspects  of  survey  methodology.  Chapter  III  is  the 
presentation  and  discussion  of  the  survey  created  by  this  project  using  the  survey 
methodology  discussed  in  Chapter  II.  Chapter  IV  is  an  initial  analysis  of  data  collected 
during  the  survey  pilot  test  and  recommendations  for  improving  the  survey  for  follow-on 
research  projects.  Chapter  V  concludes  with  the  summary  and  implications  of  the 
analysis  and  recommendations  for  further  study  in  the  area  of  the  management  of  services 
acquisition  within  the  Department  of  Defense. 
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II.  SURVEY  METHODOLOGY 


A.  INTRODUCTION 

Surveys  of  all  types  have  been  and  continue  to  be  widely  used  in  virtually  all 
areas  of  public  policy.  In  the  United  States  the  most  notable  survey  is  the  decennial 
census  conducted  by  the  U.S.  Census  Bureau.  In  addition  to  conducting  the  decennial 
census,  the  U.S.  Census  Bureau  conducts  over  100  other  survey  every  year  (U.S.  Census 
Bureau,  2007c).  The  other  surveys  carried  out  by  the  U.S.  Census  Bureau  focus  on 
numerous  areas  focused  on  demographic  or  economic  characteristics.  Demographic 
surveys  are  conducted  pertaining  to  areas  such  as  housing;  fishing,  hunting,  and  wildlife- 
associated  recreation;  property  owners  and  managers;  income;  and  women-  and  minority- 
owned  businesses  (U.S.  Census  Bureau,  2007a).  Economic  surveys  are  conducted 
regarding  areas  such  as  construction,  foreign  trade,  manufacturing,  mining,  retail, 
wholesale  and  services  (U.S.  Census  Bureau,  2007b).  The  purpose  of  these  surveys  is  to 
collect  generic  statistical  infonnation  from  individuals  and  establishments  within  the 
respective  categories  and  use  the  resulting  statistics  to  shape  public  policy.  (U.S.  Census 
Bureau,  2007c). 

There  are  numerous  types  of  surveys,  each  with  a  different  purpose.  Despite  the 
variability  in  survey  types,  each  survey  should  follow  some  basic  design  steps  and 
implement  measures  to  protect  respondents.  The  basic  steps  in  designing  a  survey  are  1) 
define  the  objectives  for  the  survey  (Fowler,  Jr.,  1984),  2)  determine  how  often  the 
survey  will  be  administered,  3)  generate  questions  and  completion  instructions  for 
respondents,  4)  plan  for  how  the  data  will  be  analyzed,  5)  conduct  a  pilot  test,  and  6) 
report  the  results.  (Fink,  2006) 

B.  TYPES  OF  SURVEYS 

Surveys  are  just  one  method  of  collecting  infonnation  to  help  describe,  compare 
or  explain  various  levels  of  knowledge  both  individual  knowledge  and/or  societal 
knowledge.  There  are  two  basic  types  of  surveys — interviews  and  self-administered. 
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Surveys  can  take  one  of  a  couple  formats  within  each  of  these  types.  An  interview  type 
of  survey  can  present  the  questions  of  the  survey  to  the  respondent  via  either  the 
telephone  or  in-person.  The  respondents  answers  are  recorded  and  analyzed  at  a  later 
dater  (Fink,  2006) 

Self-administered  surveys  on  the  other  hand  can  be  mailed  to  the  respondent  for 
completion  and  then  returned  to  the  administering  authority.  This  format  of  survey  can 
be  conducted  on-site  at  a  centralized  location,  such  as  a  public  health  clinic.  Increasingly 
self-administered  surveys  are  being  conducted  on-line.  The  underlying  characteristic  of 
self-administered  surveys  is  that  the  respondent  is  able  to  complete  the  survey  alone  and 
in  private  or  with  the  assistance  of  another  person,  other  than  an  interviewer.  (Fink, 
2006) 

The  project  team  elected  to  create  a  survey  as  this  would  be  the  best  method  for 
gathering  the  necessary  infonnation.  The  mode  of  data  collection  is  a  web-based,  self- 
administered  type  of  survey  in  order  to  collect  data  to  support  this  and  follow-on  projects. 
The  necessity  of  utilizing  a  survey  was  to  1)  standardize  measurement  of  responses,  and 
2)  attain  a  greater  level  of  granularity  than  that  currently  available  in  government-wide 
level  reports  (Fowler,  Jr.,  1984).  Additional  justifications  for  this  survey  format  were  the 
cost  savings  by  1)  not  having  to  mail  the  survey  and  postage-paid  return  envelopes  to 
over  100  Army  and  Marine  Corps  CONUS  installation  and  2)  the  existence  of  a  corporate 
account  with  the  web-based  survey  engine  SurveyMonkey  at  the  Naval  Postgraduate 
School. 


C.  ADVANTAGES  AND  DISADVANTAGES 

Each  survey  type  brings  with  it  its  own  characteristics,  advantages  and 
disadvantages.  Table  4  outlines  a  comparison  of  the  various  survey  types  discussed 
above.  The  table  highlights  the  basic  characteristic  of  each  survey  type,  the  advantages 
and  disadvantages  of  each,  along  with  special  needs  considerations  and  cost  elements  for 
each  type. 
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Table  4.  Survey  comparison  (From  Fink,  2006) 


j  Self-Administered  j 

Mailed 

On-stie 

Online 

Characteristics 

Paper  and  pencil 

Paper  and  pencil 

Internet  based 

Can  reach  large 
geographic  areas 

Information  is 

obtained 

immediately 

Worldwide 

People  are  used  to 
completing  paper 
and  pencil  surveys 

Questions  about 
survey  can  be  asked 
by  respondents  as 
they  arise 

Order  of  question 
can  be  programmed 

Advantages 

Can  take  the  survey 
with  you  and 
complete  it 
anywhere 

In  some  cases, 
surveys  can  be  done 
with  groups  of 
people 

Only  "legal" 

answeres  are 

accepted 

Can  give 
respondents  links 
that  explain 
unfamiliar  words 
and  help  with 
difficult  questions 

Data  are 
automatically 
entered  and  be 
automatically 
analyzed 

Need  a  motivated 
sample  to  return 
survey.  Many 
people  think  they 
have  too  much  to  do 
without  also  having 
to  complete  surveys 

Limited  to  responses 
from  just  those  who 
are  on  site 

Need  reliable  access 

to  Internet 

Disadvantages 

Respondents  must 
be  able  to  read,  see, 
and  write 

Respondents  must 
be  able  to  read,  see, 
and  write 

Respondent  must  be 
able  to  use  a 

browser 

Browser  must 
suppot  survey 
graphics 

System  can  go  down 
or  be  unreliable 

Up-to-date  address 
list 

Space  and  privacy 
for  respondent  to 
complete  the  survey 

Technical  expertise 

Special  needs 

Follow-up  mailings 

Convincing  method 
of  ensuring  privacy 
and  confientiality 

Incentives 

Costs 

Printing,  paper, 
envelopes,  stamps, 
incentives 

Printing,  paper, 
incentives,  survey 
supervisor,  and 
possibly  space  for 
respondent  to  work 

Mainly  technical 
(e.g.,  someone  who 
is  experience  in 
designing  online 
surveys) 

\  Interviews  [ 

Telephone 

In-person 

Can  be  done  with 
written  script  or 
computer  assisted 

Can  be  doen  with  a 
written  script  or 
computer  assisted 

Can  explore  answers 
with  respondents 

Same  as  telephone 

Can  assist 
respondent  with 
unfamiliar  words 

Need  trained 

interviewers 

Need  trained 

interviewers 

Need  to  make  sure 
respondent  is  home 

Must  find  a  suitable 
place  to  conduct 
interview 

If  using  computer- 
assisted  interviews, 
will  need  technical 
expertise  to  program 
them 

Up-to-date  phone 
numbers 

If  on-site,  need 
space  and  privacy 

Schedule  for 

reaching 

respondents 

May  be  difficult  or 
dangerous  to  go  to 
person's  home 

Many  need  a 
sampling  expert  for 
random  digit  dialing 

Incentives 

Training,  incentives, 
telephones  and 
telephone  charges, 
computers  and 
technical  expertise, 
sampling  expert, 
incentives 

Trainging,  space, 
travel,  incentives 

1.  Advantages  of  Web-based,  Self-administered  Questionnaire 


A  web-based,  self-administered  questionnaire  has  several  benefits  over  other 
survey  research  methods.  The  first  benefit  of  the  selected  survey  format  is  the  ease  of 
presentation  with  visual  and  or  audio  aids  (Fowler,  Jr.,  1984).  Survey  designers  with  the 
use  of  computers  may  incorporate  visual  and  audio  aids  to  further  assist  the  survey 
respondent  in  understanding  terms  or  questions  they  don’t  fully  understand  (Fink,  2006). 
A  second  benefit  is  the  potential  to  ask  questions  with  lengthy  or  complex  response 
choices.  Self-administered  surveys  allow  the  respondents  to  take  their  time  to  read  each 
question  and  consider  all  of  the  possible  choices  without  feeling  the  pressure  to  provide  a 
response  to  a  person.  An  additional  benefit  is  the  ease  of  asking  numerous  questions  that 
are  similar.  A  fourth  benefit  of  using  a  self-administered  questionnaire  is  the  respondent 
does  not  have  to  feel  uncomfortable  in  providing  answers  or  opinions  to  another  human 
in  face-to-face  contact  such  as  is  present  with  in-person  interviews.  (Fowler,  Jr.,  1984) 

2.  Disadvantages  of  Web-based,  Self-administered  Questionnaire 

The  election  of  a  web-based,  self-administered  questionnaire  does  bring  several 
disadvantages  as  well.  The  first  is  the  level  of  care  and  attention  demanded  by  question 
design.  Without  sound  question  design  the  respondent  may  not  respond  to  the  question 
or  may  provide  an  inaccurate  response  due  to  personal  interpretation.  Another 
disadvantage  to  this  type  of  survey  is  the  lack  of  having  a  researcher  on  site  to  supervise 
the  completion  of  the  survey,  answer  any  questions  respondents  may  have,  or  the  quality 
of  responses  given.  (Fowler,  Jr.,  1984).  A  third  disadvantage  is  in  direct  contrast  to  the 
advantage  of  being  able  to  incorporate  visual  and  audio  aids  in  the  presentation  of  the 
survey.  If  the  survey  design  team  incorporates  visual  and  audio  aids  risk  comes  in  due  to 
not  all  respondents  having  access  to  the  same  software  programs  or  computer  hardware 
(Fink,  2006). 

D.  PILOT  TEST 

Arguably  the  most  important  aspect  of  survey  design  is  the  conduct  of  a  pilot  test. 
A  pilot  test  is  the  trial  run  of  issuing  the  survey  and  collecting  response  data  (Fink,  2006). 
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One  of  the  key  survey  aspects  being  tested  during  this  crucial  methodology  step  is  to 
ensure  questions  are  written  well.  In  addition  the  pilot  test  is  a  trial  run  of  the 
appropriateness  and  applicability  of  survey  questions  in  relation  to  the  sample  audience. 
Closely  linked  to  ensuring  the  clarity  and  quality  of  questions  is  to  verify  if  the  collected 
data  will  provide  the  necessary  information.  Additionally,  the  pilot  test  provides  insight 
into  how  consistent  the  collected  data  will  be.  (Fink,  2006).  A  properly  conducted  pilot 
test  will  prove  invaluable  in  carrying  out  a  reliable  and  valid  survey  to  the  sample 
audience. 

Receiving  accurate  and  sufficient  information  about  the  survey  characteristics 
being  examined  during  a  pilot  test  relies  heavily  on  the  response  rate.  The  response  rate 
is  the  percentage  or  portion  of  completed  surveys  received  with  respect  to  the  total 
surveys  issued  (Fink,  2006).  The  recommended  minimal  response  rate  is  between  50  and 
60  percent  (Babbie,  2001  as  referenced  in  Ruane,  2005). 

Figure  1.  Response  Rate  Formula. 

„  Number  of  completed  surveys  received  . , .  , 

Response  rate  = - - - - - *100% 

Number  of  surveys  issued 


Self-administered  surveys  typically  have  lower  preliminary  response  rates — less 
than  30  percent  (Ruane,  2005).  Some  potential  reasons  for  a  low  response  rate  are  1)  the 
survey  does  not  reach  participants  and  therefore  they  cannot  complete  it,  2)  participants 
refuse  to  complete  the  survey,  and  3)  participants  are  unable  to  answer  the  survey  due  to 
illness  or  language  barriers  (Fink,  2006). 

E.  RELIABILITY  AND  VALIDITY 

Reliability  and  validity  are  very  important  intertwined  characteristics  for  a  survey. 
Without  either  of  theses  key  characteristics,  the  work  that  went  into  designing  and 
performing  a  survey  may  have  been  for  naught.  A  valid  survey  will  always  be  a  reliable 
one  but  that  same  reliable  survey  may  not  always  be  valid  for  every  instance  (Fink, 
2006). 
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An  example  of  this  link  between  reliability  and  validity  is  provided  in  the 
following  scenario.  A  hospital  administrator  is  consistently  asked  how  many  patient  beds 
are  in  a  certain  ward  of  the  hospital.  If  the  administrator  provides  the  same  answer  each 
time  the  survey  is  administered,  the  survey  is  said  to  be  reliable.  However,  if  the 
researchers  and  survey  presenters  claim  the  same  survey  question  provides  information  as 
to  the  level  of  quality  of  medical  care,  the  validity  of  the  survey  would  then  be  called  into 
question.  (Fink,  2006) 

1.  Reliability 

A  survey  necessity,  regardless  the  format,  is  the  characteristic  of  being  reliable.  A 
reliable  survey  is  one  that  provides  consistent  measures  of  important  characteristics 
despite  underlying  changes  in  the  target  audience.  Underlying  changes  in  the  target 
audience  refer  to  changes  in  experiences,  restfulness,  anger,  and  tension  at  each 
respective  time  the  survey  was  completed  by  respondents.  (Fink,  2006)  There  are 
multiple  means  to  check  for  reliability. 

a.  Multiple  Forms 

The  multiple-form  means  of  assessing  reliability  should  be  utilized  when 
the  principal  investigators  will  only  have  access  to  the  target  audience  once.  Under  this 
method  information  pertaining  to  a  measure,  such  as  a  respondent’s  age,  is  obtained  by 
asking  two  separate  questions.  One  of  the  questions  regarding  age  may  ask  the 
respondent  to  provide  their  age.  The  other  question  might  ask  the  respondent’s  year  of 
birth.  If  the  answers  to  these  individual  questions  are  consistent,  then  reliability  of  the 
measure  is  achieved.  (Ruane,  2005) 

b.  Split-half  Technique 

The  split-half  technique  is  used  when  the  measure  being  examined  is 
composed  of  multiple  aspects.  This  technique  is  a  means  of  inspecting  the  individual 
aspects  and  if  the  individual  aspects  consistently  and  equally  contribute  to  the  composite 
measure.  The  implementation  of  this  method  is  with  a  battery  of  questions  in  which  one 
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half  of  the  list  is  considered  as  a  mini-list  and  the  other  half  is  considered  as  another 
mini-list.  If  a  comparison  of  the  two  mini-lists  yields  a  high  correlation,  the  entire  list  of 
questions  is  said  to  be  reliable  for  assessing  the  measure. 

2.  Validity 

Just  as  there  were  multiple  means  of  checking  for  reliability,  there  are  multiple 
means  of  ensuring  validity.  There  are  four  validity  techniques — face,  content,  criterion 
and  construct — survey  designers  can  use  to  establish  and  test  for  validity  in  their  survey 
measures  (Ruane,  2005).  Each  of  these  respective  validity  techniques  is  discussed  below. 

a.  Face  Validity 

Face  validity  techniques  simply  ask  if  the  measure  appears  to  be  okay  or 
sound  okay.  Face  validity  assessment  is  subjective  in  nature  and  therefore  can  sometimes 
be  called  into  question.  An  example  of  questionable  face  validity  is  connecting  a  fear  of 
crime  to  a  question  that  simply  asks  about  one’s  feeling  towards  walking  at  night  alone 
(Ruane,  2005) 


b.  Content  Validity 

Content  validity  is  important  when  the  topic’s  definition  is  multifaceted 
and  complex.  Content  validity  is  a  subjective  assessment  made  by  the  researcher  whether 
the  measurement  captures  all  of  the  facets  of  the  complex  problem.  (Ruane,  2005)  Often 
times  it  is  difficult  to  capture  the  full  meaning  of  the  multifaceted  and  complex  problem 
with  just  a  single  measurement  or  question. 

c.  Criterion  Validity 

Criterion  validity  is  not  a  subjective  assessment  of  a  measure  but  rather 
uses  empirical,  objective  data  to  explain  the  measure’s  validity.  There  are  two  dominant 
strategies  to  use  when  checking  for  criterion  validity — predictive  and  concurrent.  If 
using  predictive  validity  the  principal  investigators  show  the  measurements  validity  when 
the  measurement  accurately  forecasts  a  related  outcome.  Concurrent  validity  on  the  other 
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hand  attempts  to  prove  one  measure’s  validity  by  obtaining  similar  measurements  with 
different  tests  of  the  same  concept.  (Ruane,  2005) 

d.  Construct  Validity 

Construct  validity  is  probably  the  most  difficult  of  validity  checks  and 
involves  theory  and  hypothesis  testing.  Theories  are  used  to  create  hypotheses  which 
predict  expected  relationships  between  a  measure  and  other  variables.  To  claim  construct 
validity  support  for  the  hypothesis  has  to  be  shown.  Construct  validity  demands  many 
man-hours  and  can  be  smaller,  stand-alone  research  projects  in  and  of  themselves. 
(Ruane,  2005) 

F.  ETHICS,  PRIVACY,  AND  CONFIDENTIALITY 

A  basic  ethics  guideline  in  all  research  involving  human  participants  is  the 
implementation  of  various  measures  to  help  ensure  that  no  hann  comes  to  any  participant 
(Fowler,  Jr.,  1984).  In  addition  to  this  “no  brainer”  ethical  guideline,  additional  measures 
which  must  be  implemented  by  researchers  conducting  research  involving  human 
subjects  at  agencies  which  receive  federal  funding  are  outlined  in  Part  46,  Title  45  of  the 
Code  of  Federal  Regulations.  Some  of  these  additional  measures  include  attaining 
informed  consent  from  participants  and  approval  from  an  Institutional  Review  Board 
(DHHS). 

1.  Informed  Consent 

Infonned  consent  refers  to  the  participant  determining  themselves  whether  to 
participate  in  the  research  or  not  (Ruane,  2005).  Ruane  (2005)  provides  a  sound 
discussion  of  four  characteristics  of  informed  consent  and  they  are  competence — the 
ability  of  the  participant  to  decide  whether  to  participate  or  not;  voluntarism — the 
participant’s  ability  to  freely  decide,  without  coercion  or  threat  of  retribution,  to 
participate  or  not;  full  information — the  right  of  the  participant  to  be  fully  informed  of  all 
aspects  of  the  research;  and  comprehension — the  participant’s  ability  to  understand  all 
information  given  to  them. 
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The  Code  of  Federal  Regulations  lists  eight  general  requirements  for  informed 
consent.  Each  of  these  general  requirements  needs  to  be  presented  to  the  perspective 
participant  so  that  they  can  consider  all  factors  in  their  decision  to  participate  or  not.  The 
eight  general  requirements  are  1)  a  statement  that  the  project  involves  research,  the 
purpose(s)  of  the  research,  anticipated  duration  of  participation,  a  description  of  the 
procedures,  even  experimental  ones,  that  are  being  followed,  2)  description  of  foreseeable 
risks,  3)  description  of  benefits  to  the  participant,  4)  procedures  or  other  courses  of 
treatment  that  might  benefit  the  participant  more,  5)  extent  of  measures  to  maintain 
participant  confidentiality,  6)  description  of  compensation  for  research  involving  more 
than  minimal  risk  to  the  participant,  7)  provide  points  of  contact  for  answering  questions 
or  addressing  concerns,  and  8)  a  statement  that  participation  in  the  research  is  purely 
voluntary.  (DHHS) 

2.  Institutional  Review  Board  (IRB) 

The  Institutional  Review  Board  (IRB)  concept  is  in  place  to  further  provide 
protection  to  potential  human  research  subjects.  The  implementation  of  an  (IRB)  is 
mandated  by  the  Code  of  Federal  Regulation  for  all  agencies  conducting  research  and  in 
receipt  of  federal  funding.  The  simplified  purpose  of  an  IRB  is  to  review  research 
proposals  to  ensure  that  the  researchers  have  incorporated  the  other  mandatory 
stipulations  of  45  CFR  46,  such  as  all  the  elements  of  infonned  consent  being  provided  to 
each  participant.  The  IRB  does  possess  the  authority  to  waive  certain  aspects  of  45  CFR 
46  such  as  the  researchers  receiving  a  signed  infonned  consent  form  from  each 
participant  when  it  can  be  demonstrated  that  the  research  effort  could  not  practicably  be 
completed  without  the  requirement  waiver. 

An  IRB  is  granted  certain  authorities  by  45  CFR  46.  The  first  of  which  is  the 
authority  approve  research  proposals  when  there  is  a  majority  amongst  the  board 
members.  In  addition  to  approving  research  proposals,  the  IRB  is  may  mandate  certain 
changes  to  the  research  proposal  prior  to  granting  their  approval.  Once  approved  the 
researchers  may  not  deviate  from  the  approved  protocol.  If  changes  to  the  protocol  need 
to  be  made,  the  principal  investigators  need  to  submit  an  updated  protocol  to  the  IRB  for 
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consideration.  In  a  case  where  an  IRB  disapproves  a  research  proposal,  a  reviewing 
authority  may  not  then  approve  the  research.  The  IRB’s  decision  is  final.  (DHHS) 
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III.  SERVICES  ACQUISITION  MANAGEMENT  SURVEY 

DESIGN 

A.  INTRODUCTION 

In  the  last  chapter  a  general  overview  of  survey  research  methodology  was 
provided.  Based  on  the  advantages  and  disadvantages  of  various  survey  types  the  project 
team  elected  to  conduct  a  web-based,  self-administered  questionnaire  in  order  to  gather 
data  regarding  the  current  state  of  services  acquisition  management,  at  the  installation 
level,  for  the  Army  and  Marine  Corps.  This  chapter  presents  the  methods,  justifications 
and  logic  that  were  used  to  create  the  web-based,  self-administered  questionnaire  within 
the  framework  of  survey  research  methodology. 

B.  QUESTIONNAIRE  DESIGN 

The  questionnaire  was  designed  to  collect  empirical  data  to  stratify  the 
participants  by  military  branch  of  service,  region  within  that  service,  as  applicable,  and 
by  the  particular  installation.  The  steps  taken  in  designing  this  survey  are  the  same  as 
discussed  in  Chapter  II  and  they  are  1)  define  the  objectives,  2)  determine  how  often  the 
survey  will  be  administered,  3)  generate  questions  and  completion  instructions  for 
respondents,  4)  plan  for  how  the  data  will  be  analyzed,  5)  conduct  a  pilot  test,  and  6) 
report  the  results. 

1.  Objectives  Defined 

The  objectives  of  the  survey  are  directly  linked  to  our  primary  research  question: 
What  is  the  current  state  of  services  acquisition  management  at  the  installation  level?  In 
order  to  gain  insight  into  the  current  state  of  services  acquisition  management  the  survey 
first  needs  to  collect  data  and  infonnation  on  the  acquisition  of  services  within  the  seven 
selected  service  categories.  In  addition  to  providing  infonnation  on  the  types  of  services 
contracted  the  survey  needs  to  gather  information  on  the  acquisition  strategies  and 
procurement  methods  used  to  acquire  the  services.  The  third  objective  of  the  survey  is  to 
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provide  insight  into  what  organizational  and  management  structures  are  used  and  how  do 
they  manage  the  contracts  for  services  within  the  selected  service  categories.  The  final 
objective  of  the  survey  is  to  provide  insight  into  what  types  of  training  the  acquisition  and 
project/program  management  team  receives  at  each  installation. 

2.  Survey  Duration 

This  survey  was  designed  to  be  issued  once  in  order  to  attain  a  cross-sectional 
snapshot  of  services  acquisition  management  for  each  of  the  services.  Even  though  this 
project  only  covers  Anny  and  Marine  Corps  CONUS  installations,  follow-on  research 
projects  would  use  this  same  survey  to  gain  data  and  infonnation  for  the  other  military 
services.  By  using  the  same  core  survey  the  follow-on  projects  will  be  able  to  equally 
compare  their  results  with  the  results  of  this  project,  therefore  offering  greater  insight  to 
other  researchers  as  to  the  acquisition  management  of  services  within  DoD. 

3.  Informed  Consent  and  Completion  Instructions 

A  key  component  to  conducting  a  survey  is  providing  the  required  information 
about  the  research  project  to  potential  respondent  so  they  can  make  an  informed  consent 
statement.  In  designing  our  survey  with  the  use  of  SurveyMonkey  our  first  page  is  the 
“Participation  Statement.”  In  this  area  the  potential  respondent  is  informed  that  the 
survey  is  part  of  a  research  project  to  help  identify  trends  and  best  practices  pertinent  to 
the  acquisition  of  services  at  the  installation  level.  Respondents  are  also  informed  that 
the  survey  is  completely  voluntary  and  are  provided  with  an  estimated  involvement  time. 
A  key  component  to  this  area  is  the  conveyance  that  confidentiality  and  privacy  will  be 
maintained  by  not  only  the  project  team  but  the  institution  as  well.  Lastly,  the  respondent 
is  provided  contact  information  for  the  project  team  and  the  IRB  so  that  their  questions 
and  concerns  could  be  handled  efficiently  and  effectively.  Including  all  of  the 
information  in  the  Participation  Statement  section  the  team  maintained  compliance  with 
45  CFR  46. 

Once  the  respondent  elects  to  participate  in  the  survey  the  next  webpage  seen  is 
the  “Introduction.”  On  this  webpage  the  purpose  of  the  project  is  again  conveyed  to  the 
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participant  along  with  the  survey’s  completion  instructions.  Participants  are  instructed  to 
answer  the  questions  to  the  best  of  their  ability  and  are  again  informed  that  no  personal 
information  will  be  asked  or  recorded  during  the  conduct  of  the  survey.  Infonnation  on 
how  to  move  back  and  forth  throughout  the  web-based  questionnaire  is  also  provided. 
One  of  the  key  features  of  using  a  commercial  survey  engine  is  the  respondent  does  not 
need  to  complete  the  survey  in  one  sitting.  If  the  respondent  needs  to  leave  the  survey 
site  for  any  reason,  the  responses  already  selected  will  be  saved  and  the  survey  will 
continue  from  the  point  where  the  respondent  exited  the  website.  Even  though  responses 
are  saved  for  the  respondents,  no  response  information  is  forwarded  to  the  researchers 
until  the  respondent  selects  the  “Done”  button  following  the  final  question  of  the  survey. 

4.  Question  Design 

The  questions  of  the  survey  are  organized  into  one  of  four  groups — 
administrative,  core,  general  and  comments — for  both  ease  of  design  and  flow  for  the 
respondent.  The  administrative  group  of  questions  is  presented  to  the  respondent  first 
which  helps  identify  the  military  service,  geographical  region,  and  individual  installation 
being  represented  by  the  participant.  In  this  and  other  sections  of  the  web-based 
questionnaire,  the  project  team  utilized  filter  questions  and  a  “skip-logic”  tool  provided 
by  SurveyMonkey,  whereby  the  participant’s  responses  to  earlier  questions  indicated  the 
roadmap  that  would  be  followed  through  the  rest  of  the  questionnaire.  The  combination 
of  these  two  techniques  allowed  the  respondent  to  complete  the  questionnaire  more 
rapidly  by  not  having  to  read  inapplicable  questions.  The  two  middle  groups  of 
questions — core  and  general — are  more  complex  and  are  discussed  in  further  detail  next. 

a.  Core  Questions 

The  core  questions  request  responses  for  each  of  the  seven  selected  service 
categories  and  are  further  broken  down  into  the  following  sub-groups:  contract 
characteristics,  acquisition  management  methods,  project  team  approach  and  services 
acquisition  leadership.  The  individual  questions  for  each  subject  service  category  are 
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grouped  together  to  facilitate  ease  of  use  for  the  respondent  and  so  the  respondent  could 
think  about  and  respond  for  one  service  category  at  a  time. 

(1)  Contract  Characteristics.  The  questions  in  this  sub-group 
pertain  to  each  of  the  seven  selected  service  categories.  The  survey  again  identifies  the 
seven  service  categories  considered  by  the  questionnaire  and  provides  a  basic  roadmap 
for  this  segment  of  the  survey.  The  purpose  of  these  questions  is  to  gain  insight  into  the 
dominant  type  of  contract  being  used  in  the  acquisition  of  services  at  the  installation 
level.  Answers  to  these  questions  will  also  provide  information  for  each  service  category 
of  whom — government  or  contractor — typically  bears  the  risk  associated  with  the 
contract  and  if  there  is  enough  competition  for  the  services.  The  respondents  are  to  base 
their  selection  on  the  dominant  characteristic,  for  each  service  category,  used  in  each 
fiscal  year  between  2002  and  2006.  The  characteristics  examined  in  this  section  are 
competition  (competitively  bid  or  sole  source);  contract  type  (fixed-price  or  cost-type); 
and  incentive/award  (incentive  fee  or  award  fee  or  award  tenn). 

(2)  Acquisition  Management  Methods.  The  questions  in  this  sub¬ 
group  also  pertain  to  each  of  the  seven  selected  service  categories.  The  purpose  of  these 
questions  is  to  gain  insight  into  the  types  of  management  structures  being  used  at 
installations.  The  respondent  was  asked  to  reply  based  on  the  dominant  services 
acquisition  management  method  used  at  their  respective  installation  for  the  various 
phases  of  the  acquisition  process.  The  acquisition  process  phases  are  acquisition 
planning,  solicitation,  source  selection,  and  contract  administration.  For  each  of  these 
phases  the  participant  identified  whether  the  phase  was  conducted  at  a  regional, 
installation,  or  some  other  organizational  level.  An  additional  question  in  this  sub-group 
led  respondents  to  one  of  the  next  two  questionnaire  segments.  The  question  asked  the 
participant  to  identify  whether  a  project  team  approach  was  typically  used  in  the 
acquisition  of  the  respective  service  category  at  the  installation  level. 

(3)  Project  Team  Approach.  The  questions  in  this  sub-group 
pertained  only  to  those  that  identified  a  project  team  approach  in  the  acquisition 
management  method  described  above.  If  a  project  team  approach  was  not  selected  in  this 

section,  SurveyMonkey  logic  allowed  the  participant  to  move  onto  the  next  sub-group  of 
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questions.  The  respondent  was  asked  to  identify  the  billet  of  the  project  team  leader, 
such  as  a  Program/Project  Manager  or  Contracting  Officer.  A  related  question  presented 
to  participants  was  who,  by  billet  or  organizational  level,  generated  and  approved 
changes  to  the  requirements  for  service  contracts.  The  purpose  of  these  questions  is  to 
provide  insight  into  what  types  of  contracted  services  typically  use  a  project  team 
approach  and  to  further  the  understanding  of  acquisition  management  methods  used  to 
manage  services  at  the  installation  level. 

(4)  Service  Acquisition  Leadership.  The  questions  in  this  sub¬ 
group  pertained  to  all  seven  service  categories  if  the  participant  indicated  that  the  project 
team  approach  was  not  dominantly  used  for  that  acquired  service.  For  the  applicable 
service  categories,  the  participant  was  asked  who,  such  as  a  Project/Program  Manager  or 
Contracting  Officer,  led  the  acquisition  of  that  service  at  their  installation.  The 
participant  was  also  asked  who,  by  billet  or  organizational  level,  generated  and  approved 
changes  to  the  requirements  for  service  contracts.  Again  the  purpose  of  these  questions  is 
to  gain  insight  into  what  types  of  contracted  services  typically  do  not  use  a  project  team 
approach  and  to  further  the  understanding  of  acquisition  leadership  methods  used  at  the 
installation  level. 

b.  General  Acquisition  Management  Methods  Questions 

The  general  acquisition  management  methods  questions  call  for  a  response 
pertaining  to  service  acquisition  methods  in  general  for  the  respondent’s  respective 
installation.  The  focus  of  this  question  group  is  a  battery  of  12  Likert-scale  type 
statements.  The  respondent  is  asked  to  indicate  their  level  of  agreement  of  with  each 
statement.  Possible  levels  of  agreement  are:  strongly  disagree,  disagree,  neutral,  agree 
and  strongly  agree.  Other  questions  in  this  group  ask  the  participant  about  the  types  of 
training  received  by  contract/acquisition  staff,  contractor  surveillance  and  the  length  of 
time  contract/acquisition  staff  members  serve  in  their  billets. 

The  last  group  of  questions  is  presented  to  all  respondents  and  offers  the 
respondent  the  opportunity  to  make  any  general  comments  or  provide  other  feedback 
regarding  the  topic  of  services  acquisition.  This  is  an  important  aspect  of  the  survey  to 
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facilitate  the  collection  of  additional  infonnation  and  data  that  may  not  have  been 
captured  within  the  body  of  the  questionnaire.  In  addition  the  general  comment  block 
allowed  the  participant  the  opportunity  to  voice  their  concerns  or  make  recommendations 
concerning  the  wording  of  individual  questions,  the  length  of  time  to  complete  the 
survey,  or  to  provide  other  miscellaneous  information. 

5.  Data  Analysis  Plan 

The  plan  for  analyzing  data  is  to  use  various  statistical  tools  to  conduct  an  initial 
analysis  of  the  data  to  help  identify  any  trends  or  points  of  interest  in  the  following 
categories:  1)  intra -region,  intra-military  service,  2)  intra-region,  inter-military  service, 
3)  inter-region,  intra-military  service,  4)  inter-region,  inter-military  service,  and  5)  inter¬ 
military  service. 

C.  IRB  PROCESS 

The  project  team  submitted  a  Protection  of  Human  Subjects  package  to  the  Naval 
Postgraduate  School’s  IRB  in  accordance  with  the  Naval  Postgraduate  School  Instruction 
3900.4:  Protection  of  Human  Subjects  (NAVPGSCOLINST  3900.4)  for  consideration. 
The  package  included  a  description  of  the  survey  (i.e.  anonymous,  web-based,  and  self- 
administered),  informed  consent  information,  a  request  for  waiver  of  signed  consent 
forms  due  to  the  nature  of  the  survey,  participant  completion  instructions,  and  a  copy  of 
all  survey  questions.  The  IRB  approved  our  package  to  conduct  the  pilot  test  for  the 
questionnaire. 

D.  PILOT  TEST 

The  pilot  test  period  was  from  late  October  to  mid-November.  It  was  conducted 
to  ensure  the  individual  questions  and  completion  instructions  were  well  written  and 
easily  understood  by  the  respondent.  Of  the  ten  potential  respondents  contacted,  six 
completed  the  web-based  survey  generating  a  response  rate  of  60  percent,  which  was 
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lower  than  expected.  However,  the  empirical  data  and  general  comments  provided  by  the 
respondents  prove  invaluable  in  generating  overall  survey  reliability  for  use  in  follow-on 
research  projects. 

E.  REPORT  OF  RESULTS 

Chapter  IV  presents  an  initial  analysis  of  the  empirical  data  received  from  the  five 
respondents  to  the  pilot  test.  Chapter  IV  also  presents  numerous  recommendations  for 
improving  the  questionnaire  to  1)  facilitate  follow-on  research  projects  and  2)  potentially 
increase  the  response  rate  to  help  provide  a  clear  picture  of  the  current  state  of  acquisition 
management  of  services  at  the  installation  level. 
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IV.  ANALYSIS  OF  PILOT  DATA 


A.  INTRODUCTION 

The  pilot  test  obtained  a  response  rate  of  60  percent,  or  six  completed  surveys  of 
the  possible  ten.  Although  six  results  are  not  statistically  significant,  they  do  provide 
some  insight  into  the  current  state  of  acquisition  management  of  services  at  the 
installation  level  within  the  DoD.  This  chapter  presents  an  initial  analysis  of  four  service 
categories’ — Professional,  administrative  and  management  support;  Maintenance  and 
repair  of  equipment;  Data  processing  and  telecommunications;  and  Transportation  and 
travel — data  collected  during  the  survey’s  pilot  test.  The  other  three  service  categories 
examined  during  the  survey’s  pilot  test  are  not  analyzed  due  to  several  factors  which  are 
discussed  later  in  the  chapter.  In  addition,  an  analysis  of  general  acquisition  management 
methods  is  presented.  Finally,  recommendations  for  improving  individual  questions  and 
overall  survey  design  are  provided  with  the  goal  of  facilitating  an  increased  response  rate 
during  follow-on  research  projects. 

B.  RESPONDENTS 

The  six  responses  received  were  all  from  installations  within  CONUS.  Sixty  - 
seven  percent  of  the  responses  came  from  Army  installations,  17  percent  from  Marine 
Corps  installations  and  17  percent  from  Air  Force  installations.  Eighty-three  percent  of 
responses  were  from  installations  west  of  the  Mississippi  River;  17  percent  were  from 
installations  east  of  the  Mississippi  River.  To  protect  respondent’s  privacy  and  maintain 
confidentiality  the  identification  of  individual  installation  that  responded  is  not 
provided — this  information  is  maintained  by  the  authors. 

C.  SERVICE  CATEGORIES  NOT  ANALYZED 

Three  of  the  seven  service  categories  examined  during  the  survey’s  pilot  test  did 
not  generate  enough  raw  data  or  substantial  general  comments  to  warrant  statistical 
analysis.  The  three  service  categories  not  analyzed  are  Medical  (FSC  Q),  Maintenance 
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and  repair  of  real  property  (FSC  Z)  and  Utilities  and  housekeeping  (FSC  S).  Nearly  all 
respondents  skipped  every  question  for  each  of  these  service  categories.  In  the  case  of 
Maintenance  and  repair  of  real  property  and  Utilities  and  housekeeping  the  general 
comments  supplied  indicated  that  these  service  categories  were  part  of  an  overarching 
Base  Operating  Support  (BOS)  contract.  There  were  no  general  comments  provided 
which  gave  insight  into  how  Medical  services  were  acquired. 

D.  SERVICE  CATEGORIES  ANALYZED 

The  web-based,  self-administered  survey  designed  for  this  project  was  interested 
in  seven  of  the  24  service  categories  as  discussed  in  Chapter  I.  During  the  survey’s  pilot 
test  four  of  the  seven  service  categories — Professional,  administrative  and  management 
support  (FSC  R),  Maintenance  and  repair  of  equipment  (FSC  J),  Data  processing  and 
telecommunications  (FSC  D)  and  Transportation  and  travel  (FSC  V) — presented  enough 
data  for  analysis  and  general  conclusions  regarding  the  current  state  of  services 
acquisition  management. 

1.  Contract  Characteristics 

The  vast  majority,  88  percent,  of  respondents  indicated  contracts  within  these  four 
service  categories  were  predominantly  bid  competitively  during  the  five  fiscal  year 
period  evaluated  by  the  survey.  The  remaining  12  percent  indicated  that  the  contracts 
were  sole  sourced.  Such  a  significant  majority  indicates  that  competition,  at  least  for 
these  service  categories,  is  sought  and  adequate. 

Fixed-price  contract  was  the  dominant  contract  type  used  for  acquiring  these 
services.  Seventy  percent  reported  using  fixed-price  contracts  indicating  that  contractors 
bore  the  preponderance  of  risk.  Thirty  percent  utilized  cost  type  contracts  placing  a 
higher  level  of  risk  on  the  government. 

Only  one  respondent  indicated  the  use  of  incentives  in  the  form  of  award  fees. 
Award  fees  were  only  used  in  the  acquisition  of  Maintenance  and  repair  of  equipment 
and  Transportation  and  travel  services.  As  reported  it  would  seem  that  incentives  could 

be  better  utilized  in  contracts  for  these  services,  possibly  to  drive  contractor  performance. 
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Yet  the  low  use  of  incentives  reported  also  shows  less  opportunities  for  abuse  where 
incentives  are  awarded  without  the  requisite  superior  performance  level  being  met. 


2.  Acquisition  Management  Methods 

The  data  show  that  the  acquisition  phases  for  each  of  the  service  categories 
continue  to  be  conducted  at  the  installation  level.  The  only  exceptions  to  this  trend  were: 
1)  An  Army  installation  reported  the  solicitation  and  source  selection  phases  for 
Maintenance  and  repair  of  equipment  were  completed  at  the  regional  level,  and  2)  A 
Marine  Corps’  installation  indicated  that  they  complete  all  acquisition  phases  at  the 
regional  level.  The  Marine  Corps’  responses  were  consistent  with  their  transition  to 
regional  contracting  offices  over  the  past  few  years. 

3.  Project  Team  Approach 

Although  data  indicates  a  high  percentage  use  of  project  teams  for  the  acquisition 
of  services,  none  of  the  respondents  indicated  a  program  manager  leads  the  services 
acquisition  team.  The  dominantly  identified  billet  for  leading  the  acquisition  team  was 
the  contracting  officer. 

Table  5.  Project  Team  Approach  Used. 


Is  a  I 
Team  A 
typicalh 
the  acqu 
services 
install 

’reject 
.pproach 
/  used  in 
isition  of 
;  at  your 
ation? 

Who,  on-site  (at  your 
installation),  leads 
the  acquisition  of 
services? 

Who  owns  (generates 
and  approves  changes 
to)  the  requirements  for 
service  contracts? 

FSC  Categories 

Yes 

No 

Contracting 

Officer 

Customer 

Contracting 

Officer 

Customer 

Professional, 
administrative  and 
management 

5 

1 

6 

1 

5 

Maintenance  and  repair 
of  equipment 

3 

3 

5 

1 

4 

Data  and 

telecommunications 

3 

3 

5 

1 

1 

5 

Transportation  and  travel 

4 

2 

3 

1 

3 

Total 

63% 

38% 

90% 

10% 

15% 

85% 
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Despite  the  indication  that  project  teams  are  utilized  in  the  acquisition  of  services, 
the  project  team  concept  is  not  employed  in  the  same  manner  as  in  systems  acquisition 
where  a  program  manager  leads  the  acquisition  of  the  system  and  owns  and  writes  the 
requirements  for  the  system.  The  pilot  survey  results  however  indicate  a  possible 
disconnect  between  the  contracting  officer  who  leads  the  acquisition  of  services  and  the 
customer  who  owns  and  writes  the  requirements. 

E.  GENERAL  ACQUISITION  MANAGEMENT  METHODS 

This  segment  of  the  survey  included  four  questions  on  general  acquisition 
management  characteristics  at  the  installation  level,  regardless  of  service  category.  These 
questions  were  followed  by  a  battery  of  12  statements  asking  the  respondent  to  indicate 
their  level  of  agreement  with  the  given  statement.  Possible  levels  of  agreement  were 
Strongly  Disagree,  Disagree,  Neutral,  Agree  or  Strongly  Agree.  The  indicated  levels  of 
agreement  to  some  of  these  statements  pose  several  points  of  interest. 

From  the  four  questions  on  general  acquisition  management  characteristics  one 
point  of  interest  is  the  responses  to  the  length  of  time  Contracting  Officer  Representatives 
(COR)  and  Quality-assurance  Evaluators  (QAE)  serve  in  their  billets.  Eight-three  percent 
of  COR  and  QAE  personnel  serve  in  their  billets  2  or  less  years.  The  implied  high 
turnover  rate,  especially  the  33  percent  who  serve  a  year  or  less,  can  have  negative 
impacts  on  the  quality  of  contractor  surveillance. 

Figure  2.  Time  in  Billet. 


less  than  6  months 
■  6  to  12  months 
H  12  to  24  months 
24  to  36  months 
□  over  36  months 
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High  turnover  rates  are  indicative  of  personnel  having  to  take  time  to  learn  the 
new  position  and  a  decreased  level  of  concentration  while  in  the  position  due  to  a  focus 
on  follow-on  assignments.  These  are  some  of  the  factors  that  lead  to  lower  quality 
service  contract  oversight. 

In  the  last  question  12  statements  describe  aspects  of  services  acquisition,  11  of 
which  should  be  the  normal  state  of  services  acquisition  at  the  installation  level.  First, 
four  of  these  questions  pertain  to  service  acquisition  personnel  billets,  manning  levels, 
training  and  qualification.  The  results  of  the  survey,  shown  in  Figure  3,  confirm  what 
GAO  has  been  reporting  regarding  the  services  acquisition  workforce:  that  it  was  and 
still  is  undermanned,  undertrained  and  under-qualified.  All  of  the  respondents  disagreed, 
some  strongly,  that  there  are  an  adequate  number  of  billets  for  services  acquisition 
management  at  the  installation  level.  They  also  overall  disagreed  with  the  following 
statement  of  acquisition  management  billets  at  the  installation  are  adequately  manned.  It 
seems  at  33  percent  of  installations  services  acquisition  management  staff  members  are 
adequately  trained  or  qualified,  but  at  best  they  are  not  above  50  percent.  Although  the 
results  show  nothing  new,  they  clearly  indicate  we  cannot  expect  things  to  improve  until 
the  situation  changes. 

Figure  3.  Personnel  billets,  manning,  training  and  qualification. 


□  *  There  are  an  adequate  number  of 
services  acquisition  management  stafF 
positions/billets  at  this  installation. 

■  *  Services  acquisition  management  stafF 
positions/billets  at  this  installation  are 
adequately  filled/manned. 


□  *  Services  acquisition  management  stafF 
members  at  this  installation  are 
adequately  trained. 

□  *  Services  acquisition  management  stafF 
members  at  this  installation  are 
adequately  qualified. 


Strongly  Disagree  Neutral  Agree  Strongly 
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Two  of  the  12  questions  investigate  the  use  of  a  lifecycle  approach  for  managing 
acquired  services.  Fifty  percent  of  the  respondents  disagreed  that  a  lifecycle  approach  is 
used  at  their  respective  installation  (Figure  4)  for  both  routine  and  non-routine  services. 
The  lack  of  a  lifecycle  approach  for  routine  and  non-routine  services  has  the  potential  to 
place  the  government  at  a  higher  level  of  risk  due  to  improper  planning  for  the  various 
phases  in  a  service’s  lifecycle. 

Figure  4.  Lifecycle  approach. 


Strongly  Disagree  Neutral  Agree  Strongly 
Disagree  Agree 


□  *  For  routine  services,  a  lifecycle 
approach  is  a  dominant  strategy 
used  in  the  acquisition  of  these 
services. 


■  *  ForNON-routine  services,  a 
lifecycle  approach  is  a  dominant 
strategy  used  in  the  acquisition  of 
these  services. 
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Figure  5  depicts  statements  regarding  Quality-assurance  Evaluators  including 
their  training  and  their  performance  of  duties  as  well  as  the  level  of  oversight  provided  to 
monitor  contractor  performance,  whether  conducted  by  QAE  or  other  personnel.  The 
responses  are  spread  across  the  spectrum,  but  are  more  negative  than  positive. 
Respondents  overall  agree  that  the  training  of  QAE  and  submitting  of  reports  but 
predominantly  disagreed  about  affording  a  proper  level  of  oversight  to  monitor  contractor 
performance.  This  further  confirms  what  GAO  reported  and  shows  that  changes  are  still 
needed  in  this  area  as  oversight  is  vital  to  ensuring  adequate  contractor  performance. 

Figure  5.  Training  and  oversight. 
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flE 


mr 


Strongly  Disagree 
Disagree 


Neutral  Agree  Strongly 
Agree 


□  *  Personnel  responsible  for  the 
surveillance  of  contractors  receive 
fonnal/documented  training  prior  to 
assuming  the  duties  as  a 
Quality-assurance  Evaluator  (QAE). 

□  *  Quality-assurance  Evaluators  (QAE) 
submit  written  reports  regarding  the 
performance/quality  of  work  of 
contractors  to  the  regional  contracting 
olficer/olEce  for  each  service  contract. 

□  *  Services  contracts  are  afforded  the 
proper  level  of  oversight  to  monitor 
contractor  performance. 
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Finally,  Figure  6  shows  that  the  respondents  predominantly  agree  that  there  is  no 
discrepancy  between  requirements  identification  and  Statements  of  Work/Objectives. 
Thus  the  cost  increase  is  not  due  to  miscommunication  of  requirements  and  objectives. 
Respondents  also  agreed  that  market  research  was  conducted  for  the  acquisition  of 
services. 


Figure  6.  Positive  responses. 


1 

1 

□  *  The  entity  that  identifies  the 
requirements  in  a  service  contract  also 
writes  the  Statement  of Work/Statement  of 
Objectives  (SOW/SOO)  for  the  service 
contract. 

■  *  Market  research  is  conducted  for  the 
acquisition  of  services. 


Strongly  Disagree  Neutral  Agree  Strongly 
Disagree  Agree 


F.  RECOMMENDATIONS  TO  IMPROVE  SURVEY 

The  completion  of  the  pilot  test  not  only  gathered  valuable  data  for  analysis,  it 
provided  insight  into  how  the  survey  questions  were  received  by  the  respondents  and 
whether  the  questions  were  applicable  to  the  current  state  of  services  acquisition 
management. 

The  first  survey  improvement  recommendation  is  to  provide  more  concise  and 
clear  instructions  to  participants.  The  current  completion  instructions  do  not  clearly  state 
that  a  team  approach  may  be  used  to  answer  the  survey  questions.  A  team  approach  to 
providing  answers  may  improve  the  snapshot  of  acquisition  management  of  services  for 
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that  particular  installation.  The  instructions  still  need  to  be  clear  that  each  installation 
should  only  submit  one  complete  survey  and  to  not  allow  individual  team  members  to 
submit  partially  completed  surveys. 

A  second  survey  improvement  recommendation  is  to  decrease  the  number  of 
survey  questions  presented  at  one  time.  The  survey  issued  during  the  pilot  test  contains 
85  questions,  one  of  which  has  a  subset  of  12  questions.  Although  SurveyMonkey 
provides  means  to  implement  skip  logic,  85  questions  may  be  just  too  long.  A  means  of 
shortening  the  survey  would  be  to  remove  the  three  service  categories — Medical;  Utilities 
and  housekeeping;  and  Maintenance  and  repair  of  real  property — that  did  not  generate 
enough  data  or  information  for  analysis. 

Another  survey  improvement  recommendation  is  to  create  a  survey  based  solely 
on  a  battery  of  questions  that  can  be  answered  using  a  Likert-type  scale.  During  the  pilot 
test  of  the  current  survey,  all  respondents  supplied  an  answer  for  each  of  the  12  questions. 
Future  additional  questions  should  ask  if  manning  levels  and  billet  fill  rates  have  declined 
and  the  level  of  decline  over  the  past  five  years.  This  tactic  of  asking  questions  may  be 
faster  for  the  respondent  to  complete  and  still  provide  significant  insight  into  the 
management  of  services  acquisition. 

A  fourth  survey  improvement  recommendation  is  to  add  a  question  within  each 
service  category  that  requests  quantitative  data  regarding  the  dollars  obligated  for  the 
service  category  in  each  year  for  a  range  of  fiscal  years.  The  recommended  means  to 
accomplish  this  is  to  provide  the  participant  with  several  dollar  value  ranges  and  require 
that  they  select  one  of  the  ranges  by  checking  a  box.  An  alternative  way  of  capturing 
quantitative  data  is  to  provide  a  blank  text  box  and  require  the  respondent  to  input  data. 
The  recommended  method  allows  the  survey  designer  to  choose  and  format  the  dollar 
value  ranges;  the  second  method  requires  the  survey  designer  to  clearly  convey  to  the 
participant  the  format  of  the  dollar  value  entry  (e.  g.  use  of  $  or  comma  or  whole  dollars). 

Lastly,  the  survey  should  add  a  question  asking  each  respondent  to  indicate  the 
amount  of  time  spent  to  complete  the  survey.  The  survey  design  options  for  this 
question — range  or  text  box — are  similar  to  those  for  dollars  obligated  discussed  above. 
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V.  SUMMARY,  CONCLUSIONS  AND  RECOMMENDATIONS 


A.  SUMMARY 

The  goal  of  this  project  was  to  gain  insight  into  the  current  state  of  acquisition 
management  of  services  at  the  installation  level  and  how  to  best  collect  empirical  data  in 
this  area  of  study.  Chapter  II  provided  an  overview  of  survey  methodology  and  based  on 
the  advantages  and  disadvantages  of  various  survey  types,  a  web-based,  self-administered 
survey  was  deemed  the  best  method  to  collect  the  desired  empirical  data.  Chapter  III  was 
a  discussion  of  the  various  survey  design  aspects  of  the  survey  presented  during  the  pilot 
test.  Chapter  IV  analyzed  the  pilot  test  data  for  four  of  the  seven  service  categories  and 
made  several  recommendations  to  improve  the  survey  to  facilitate  follow-on  research 
projects.  Here  in  Chapter  V,  the  project’s  conclusions  and  recommendations  for  further 
study  are  presented. 

B.  CONCLUSIONS 

The  most  appropriate  research  method  to  collect  the  amount  of  desired  data  in  this 
area  of  study  is  to  conduct  a  web-based,  self-administered  questionnaire.  The  survey 
created  for  this  project  serves  as  a  baseline  for  future  research  projects.  Without  a  pilot 
test  of  a  survey  to  test  the  questions  in  a  real  world  environment,  reliable  and  valid  survey 
results  cannot  be  expected  from  a  larger  sample  of  the  installation  population.  By 
incorporating  the  recommended  design  changes  to  improve  the  survey,  further  data 
collection  efforts  will  yield  higher  quality  data  and  provide  greater  insight  into  the 
management  of  services  acquisition  at  the  installation  level  across  DoD. 

The  current  state  of  services  acquisition  management  at  the  installation  level,  as 
revealed  by  the  initial  analysis  of  survey  pilot  test  data,  demonstrates  several  of  the  key 
aspects  causing  increases  in  service  contracts.  Some  of  these  aspects  include  deficit  billet 
and  manning  levels  which  are  further  exacerbated  by  inadequate  training  and  experience 
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of  acquisition  personnel.  The  lack  of  robust  project  team  and  lifecycle  approaches  in  the 
acquisition  management  of  services  is  another  factor  contributing  to  ineffective  and 
inefficient  management. 

C.  RECOMMENDATIONS  FOR  FURTHER  STUDY 

Acquisitions  management,  whether  for  a  weapon  system  or  a  service,  contains  a 
broad  range  of  topics  deeming  multiple  opportunities  for  further  study  and  potential 
management  improvement  recommendations.  The  authors  present  some  areas  for  further 
study  in  the  management  of  services  acquisition  area. 

In  Chapter  IV  the  discussion  of  which  service  categories  were  not  analyzed 
identified  that  several  service  categories  are  sometimes  grouped  together  under  an 
overarching  Base  Operating  Support  (BOS)  contract.  Studies  focusing  on  BOS  contracts 
could  examine  the  types  of  services  typically  bundled  together  in  a  BOS  contract. 
Empirical  data  could  be  collected  to  identify  trends  in  contract  characteristics,  which 
acquisition  management  methods  are  used,  or  if  the  military  services  handle  this  type  of 
contract  differently. 

Additional  recommendations  for  further  study  focus  on  expansion.  The  first  is  to 
expand  the  geographical  boundaries  of  the  survey  by  issuing  it  to  all  CONUS  and 
OCONUS  installations  for  all  of  the  military  services.  A  worldwide  survey  would  help 
identify  trends  between  CONUS  and  OCONUS  installations  along  with  trends  between 
the  military  services. 

Other  expansion  options  include  1)  conduct  surveys  to  capture  empirical  data 
regarding  the  other  17  service  categories,  2)  expand  the  number  of  fiscal  years  considered 
to  identify  long-term  trends,  and  3)  create  a  survey  to  gather  quantitative  data  on  service 
contracts  administered  under  the  Simplified  Acquisition  Process. 

One  last  recommendation  for  further  study  in  the  management  of  services 
acquisition  is  to  examine  other  DoD  agencies  and  field  activities  such  as  the  Defense 
Logistics  Agency  (DLA),  the  Defense  Finance  and  Accounting  Service  (DFAS)  and  the 
Defense  Technical  Information  Center  (DTIC). 
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